Workshop  Session  #3: 

Human  Interaction  with  Embedded  Virtual  Simulations 
Summary  of  Discussion 


This  workshop  session  was  facilitated  by  Dr.  Thomas  Alexander  (GER)  and  Dr.  Sylvain  Hourlier  (FRA) 
and  focused  on  interface  technology  and  human  effectiveness  including  sensors  and  controls,  visual 
perception  and  decision  making  (e.g.  target  detection  and  identification)  within  EVS.  Four  papers  related 
to  EVS  interface  technology  were  developed  and  three  were  presented;  Dr.  Godroy  was  unable  to  attend 
due  to  illness. 

•  Embedded  Augmented  Reality  Training  System  for  Dynamic  Human-Robot  Cooperation  Jan  A. 
Neuhofer,  Dipl.-Ing.  (GER);  Bernhard  Kausch,  Dr.-Ing.  (GER);  and  Christopher  M.  Schlick, 
Prof.-Ing.  (GER),  Institute  of  Industrial  Engineering  and  Ergonomics,  Aachen  University 

•  Human  Dimensions  in  Multimodal  Wearable  Virtual  Simulators  for  Extra  Vehicular  Activities  - 
Martine  Godfroy,  Ph.D.  (FRA/USA),  NASA  Ames  Research  Center/  San  Jose  State  University) 

•  Embedded  Training  in  a  Ground  Soldier  System-  Jean  Dyer,  Ph.D.  (USA),  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences 

•  Interaction  between  Reference  Frames:  a  concern  in  embedded  training  -Patrick  Sandor,  M.D., 
Ph.D.  (FRA),  IRBA;  D.  Hartnagel,  Ph.D.  (FRA),  IRBA;  L.  Bringoux,  Ph.D.  (FRA),  ISM;  C. 
Bourdin,  Ph.D.  (FRA),  ISM;  M.  Godfroy,  Ph.D.  (FRA/USA),  NASA  Ames  Research  Center/  San 
Jose  State  University;  C.  Roumes,  M.D.,  Ph.D.  (FRA),  IRBA 

Workshop  Exercise  #3:  Mindmapping  EVS  interface  technology 

For  this  session,  the  mindmapping  exercise  focused  on  EVS  interface  technology  and  the  question  below 
was  intended  to  focus  and  energize  the  discussion: 

•  What  are  the  capabilities  and  limitations  of  embedded  virtual  simulation  interface  technology? 

The  mindmaps  shown  below  are  the  group  products  from  Exercise  #3  on  EVS  interface  technology.  Note 
that  the  mindmap  for  Group  1  is  missing: 
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Figure  1:  Group  2  Mindmap  for  Exercise  # 3 
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Figure  2:  Group  3  Mindmcip  for  Exercise  # 3 

There  are  just  two  mindmaps  available  (check  if  the  others  have  the  missing  ones).  They  differ  from  each 
other  at  a  basic  level.  MM1  is  more  conceptual  and  addresses  capabilities  and  limitations  of  EVS  interface 
technology.  On  the  other  hand,  MM2  addressed  special  issues  of  each  modality  for  i/o  interfaces  and 
several  more  general  issues  with  EVS  interface  technology. 

The  first  map  starts  with  addressing  some  of  the  current  capabilities  which  are  already  available.  An 
example  for  this  technology  is  the  development  of  flexible  transparent  displays,  which  can  be  used  for 
fighter  aircrafts. 


RTO-MP-HFM-1 69 


D3-3 


Workshop  Session  #3 

Human  Interaction  with  Embedded  Virtual  Simulations 


The  second  category  is  future  capabilities,  which  have  to  be  addresses  although  they  are  challenging.  They 
focus  particular  voice  recognition,  problems  of  information  output  and  input  technology  and  issues  for 
special  applications.  Voice  recognition  is  such  a  challenging  topic  because  some  applications  (e.g. 
aerospace  applications)  are  characterized  by  a  noisy  environment.  This  can  be  partly  resolved,  e.g.  as  it 
has  happened  in  case  of  the  F35.  Furthermore,  discourse  might  lead  to  wrong  interpretations.  Finally  there 
is  the  intelligibility.  With  regard  to  the  output  of  information,  some  issues  with  visual  displays  were 
addressed  as  an  example.  In  case  of  aerospace  applications,  an  HMD-like  see-through  display  would  be 
first  choice.  In  case,  training  includes  visual  tasks,  simulated,  computer-generated  forces  have  to  be 
visualized  on  the  HMD.  From  a  technical  point  of  view,  this  puts  high  demands  on  the  head  tracking  to 
allow  correct  angular  velocity  of  the  targets  for  a  consistent  information  display. 

Input  technology  requires  new  types  of  sensors  and  of  sensor -data  fusion.  Sensor  images  have  to  be 
consistant  to  establish  a  realistic  correspondence  between  each  other.  Some  more  general  issues  which 
have  to  be  addressed  with  future  challenges  are  capabilities  for  instructor  input.  An  EVS  system  has  to 
facilitate  instructors  input  throughout  training.  This  raised  the  issue  of  connecting  multiple  platforms  with 
each  other  and  integrating  white-cell  personnel  on  a  technical  level.  Another  more  general  issue  addressed 
the  simulation  of  motion  with  a  static  vehicle.  Yet,  motion  is  only  induced  by  visual  means,  which  might 
result  into  negative  effect,  e.g.  motion  sichness.  Participants  noted  that,  especially  on  C2  systems,  chat 
boxes  are  increasingly  used.  It  might  help  to  include  Web  2.0  methods  and  technologies  to  include  the 
same  into  training  systems.  This  will  enhance  interaction  and  communication  between  trainer  and  trainees. 
However,  it  is  unclear  how  this  can  be  included  into  the  EVS  concept. 

The  third  category  addresses  limitations  inherent  to  the  concept  of  EVS  which  cannot  be  resolved.  The 
address  more  general  issues  of  EVS.  In  addition  to  technical  problems  with  the  integration  of  EVS 
hardware  into  existing  systems,  i.e.  weight  and  dimensions,  it  is  unclear  whether  the  real  system  is  always 
the  best  training  system.  This  might  be  true  for  some  phases  of  training,  but  not  necessarily  for  every 
phase.  Therefore,  EVS  should  be  considered  as  a  part  of  education  and  training,  which  extends  traditional 
training  and  does  not  replace  it.  It  is  still  unclear,  how  EVS  can  be  integrated  into  existing  platforms, 
because  they  are  sometimes  highly  modular  with  no  general,  common  interface  or  bus  system.  E.g.  the  fire 
control  system  is  often  not  connected  to  the  weapon  system  and  has  to  be  operated  manually.  In  this  case, 
EVS  cannot  be  consistently  integrated.  Another  general  issue  is  the  legal  issue  of  the  EVS  system  and  the 
database.  It  is  unclear,  if  the  fully  embedded  EVS  can  be  fully  integrated  into  the  operational  system  when 
the  operational  system  is  already  fielded.  Therefore,  options  of  straped-on  EVS  have  to  be  considered 
accordingly. 

With  regard  to  limitations  and  capabilites,  the  second  mindmap  differentiates  between  the  modalities  of 
displays  and  input  interfaces. 

With  regard  to  the  modality  of  the  display,  most  information  is  perceived  by  the  visual  modality.  There  are 
various  displays  in  operation  already,  which  allows  integrating  synthetic  additional  visual  information  into 
the  sight.  Examples  of  these  are  thermal  or  night-vision  displays.  They  present  additional  information 
which  is  different  from  reality  sight.  Technologies  from  the  field  of  Augmented  Reality  (AR)  facilitate  the 
integration  of  additional  information  into  real  sight.  But  yet,  the  consistent  integration  of  geometric 
information  into  the  sight  is  a  problem  because  of  occlusion  and  cluttering  with  reality.  It  might  also 
require  a  complex  calibration  of  the  EVS  system’s  components  and  displays.  But  such  displays  are  not 
limited  to  EVS,  instead  they  might  also  be  used  to  support  the  soldier  during  mission  execution  because 
they  can  also  be  used  to  present  blue  and  red  forces.  Thus,  the  technology  is  not  stand-alone  but  cross¬ 
application. 

There  are  also  capabilities  of  presenting  EVS  information  through  the  aural  modality.  Today,  3D  sound  is 
widely  available,  which  facilitates  a  realistic  simulation  of  the  aural  environment.  But  there  is  a  large 
variability  of  human  performance.  This  might  limit  the  applicability  of  sound  in  EVS  systems.  The  aural 
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modality  can  be  used  to  direct  (and,  thus,  control)  attention.  At  this  point,  it  is  unclear  which  types  of 
sound  to  use  and  how  to  specify  aural  symbology.  Research  is  required  in  this  field  to  use  it  successfully. 

First  haptic  displays  are  currently  being  analyzed  and  examined  for  various  purposes.  Examples  are 
various  tactile  vests  and  tactile  belts,  which  can  be  used  to  display  symbolic  (haptic  icons)  and  directional 
information.  However,  these  prototypes  are  still  topic  of  research  and  it  is  unclear  how  they  can  be  used 
within  EVS  systems. 

The  relevance  of  the  kinesthetic  modality  differs  from  each  other  between  applications.  Kinesthetic 
simulation  requires  a  relatively  large  investment  and  special  technology.  For  aerospace  operations  it  is 
considered  to  be  important,  while  ground  operations  vastly  require  kinesthetic  cues.  Cross  application  it  is 
important  that  kinesthetic  cues,  either  available  or  not,  has  limitations  and  might  induce  negative  side- 
effects  like  disorientation  or  motion  sickness. 

Input  interfaces  often  require  special  sensor  technologies,  which  are  not  always  available  today.  This  is 
critical  when  operator  actions  have  to  be  captured,  e.g.  head  movements  for  a  consistent  display  of 
geometric  information.  The  benefit  of  EVS  is  that  it  can  widely  rely  on  the  existing  technology  and 
available  controls  of  the  operational  platform.  This  limits  variability,  but  enhances  realism  and  reality  of 
the  training  environment.  An  open  issue  is  the  integration  of  feedback.  E.g.  force  feedback  requires 
additional  technology  which  is  connected  to  the  real  controls. 

It  is  obvious,  that  the  use  of  multi-modal  information  presentation  requires  a  careful  integration  of  each 
display  in  order  to  prevent  a  cue  conflict  and  following  negative  side-effects. 

The  second  category  of  the  mindmap  raises  open  questions  about  general  issues.  A  very  important  one  is 
safety.  In  addition  to  certification  issues,  the  overlap  between  simulation  and  reality  is  minimized.  In  an 
ideal  EVS  system,  the  soldier  will  not  be  able  to  tell  the  difference  between  both.  Thus,  it  is  unclear  if  a 
failure  is  simulation  or  real.  Unlike  in  traditional  simulation,  EVS  has  to  include  a  method  to  present  and 
to  remind  the  soldier  on  the  mode  the  system  is  in.  The  soldier  has  to  be  aware  what  is  real  and  what  is 
simulated,  otherwise  dangerous  situation  might  appear.  This  is  especially  crucial  for  aerospace  operations 
with  high  g-loads.  EVS  cannot  be  considered  as  one  solution  for  each  application,  instead  EVS  can  serve 
as  a  method  and  means  at  special  phases  of  training.  Integration  of  EVS  always  requires  a  close 
consideration  of  its  benefits,  because  it  comes  with  costs  of  increased  weight,  size,  costs,  workload, 
fatigue  and  energy.  However,  because  some  of  its  capabilities  might  be  used  for  other  puiposes  than 
training  as  well  (e.g.  augmented  reality). 
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